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The undersigned requests that the present 

international application be processed 
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International Application 



JPCT/CN 03/ 0 10 8 7 



International Filing Date 1 3 OcC 2u03 (I 9>_XjL|0^ ) 



RO/C 




:.ijplication 



Name of receiving Office and "PCT International Application" 



Applicant's or agent's file reference 

(if desired) (12 characters maximum) P3959/BST 



Box No. I TITLE OF INVENTION 

Electronic Pen-Computer Multimedia Interactive System 


Box No. II APPLICANT This person is also inventor 


Name and address: (Family name followed by given name: for a legal entity, full official designation. 
TJte address must include postal code and name of country. The country of the address indicated in this 
Box is the applicant 's State (that is. country) of residence if no State of residence is indicated beiow.) 

\lntol Corporation? J/VTU CokpokAjIoN 
2200 Mission College Boulevard 
Santa Clara, California 95052 
United States of America 


Telephone No. 


Facsimile No. 


Teleprinter No. 


Applicant's registration No. with the Office 


State (that is. coumrv) of nationality: 

us 


Stale (that is. coitntfv) of residence: 

us 


This person is applicant l | all designated [X3n designated States except | I the United States j 1 the States indicated in 
for the purposes of: 1 1 States |^ I the United States of America | | of America only | j the Supplemental Box 


Box No. Ill FURTHER APPLICANT(S) AND/OR (FURTHER) INVENTOR(S) 


Name and address: (Family name followed by given name; for a legal entity, fill ojficial designation. 
The address must include postal code and name of country. The countty of the address indicated in this 
Box is the applicant 's State (that is. country) of residence if no State of residence is indicated below.) 

"^ntc! China Ltdj ^JNTBL ^HlNA /.TP. 
Beijing Kerry Center 

6/F, North Tower, No.1 GuangHua Road 
Chao Yang District 100020 "^^eJiyr^ 
China 


This person is: 

|X 1 applicant only 

1 1 applicant and inventor 

1 1 inventor only (If this check-box 
\ 1 is marked, do not Jtll in below.) 


Applicant's registration No. with the Office 


State (that is, countrv) of nationality: 

CN 


State (that is, country) of residence: 

CN 


This person is applicant 1 | all designated all designated States except I | the United States | 1 the States indicated in 
for the purposes of: 1 1 States the United States of Amcnca 1 1 of America only 1 1 the Supplemental Box 


y 1 Further applicants and/or (further) inventors are indicated on a continuation sheet. 


Box No. IV AGENT OR COiVIMON REPRESENTATIVE; OR ADDRESS FOR CORRESPONDENCE 


The person identified below is hereby/has been appointed to act on behalf ^ V / oaent 1 1 common 

of the applicant(s) before the competent International Authorities as: /\ ^ • 1 1 representative 


Name and address: (Family name followed by given name; for a legal entit 
The address must include postal code and name of co 


y.full official designation, 
untry.) 


Telephone No. 

86-10-88087206 


feftgxiii & PdiLiiubJ ^ l<jqA/^^V Si PAR.'^ 
Suite 2516, Xinlong Mansion 


Facsimile No. 

86-10-88087200 


Av5J bnongiu, Aicneng uistnct 

Beijing 100032 

P.R.China 


Teleprinter No. 


Agcnt*s registration No. with the Office 

11240004 


1 1 Address for correspondence: Mark this check-box where no agent or common representative is/has been appointed and the 
1 j space above is used instead to indicate a special address to which correspondence should be sent. 
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Continuation of Box No. Ill FURTHER APPLICANT(S) AND/OR (FURTHER) INVENTOR(S) 

If none of the following sitb-boxes is used, this sheet should not be included in (he request. 



Name and address: (Family namefollowed by given name: for a legal entity./itilajgriciatdesi^^ 
Vie address must include postal code and name ^country. Tlie country of the address indicated in this 
Box is the applicant 's State (that is. country) of residence if no State of residence is indicated below.) 

YEUNG, Minerva 
969 Sutter Avenue 
Sunnyvale, California 94086 



This person is: 

I I applicant only 

applicant and inventor 



□ inventor only (If this check- box 
is marked, do not fill in below.) 



United States of America 


Applicant's registration No. with theOtTice 


State (that is, countrv) of nationality: 

us 


State (that is, country) of residence: 

us 


This person is applicant j 1 all desienated | 1 all designated States except y 

for the purposes of: 1 1 States ^ I I the United States of America 


the United States j 1 the States indicated in 

of America only | | the Supplemental Box 


Name and address : (Family name follo\ved by given name: for a legal entity Jull ajficial designation. 
The address must include postal code and name oj'country. The country of the address indicated in this 
Box is the applicant 's State (that is, country) of residence if no State of residence is indicated below.) 

YEO, Boon-Lock 
969 Sutter Avenue 
Sunnyvale, California 94086 
United States of America 


This person is: 

1 [ applicant only 

[ X| applicant and inventor 

1 1 inventor only (If this check-box 

1 1 is marked, do not fill in below.) 


Applicant's registration No. with the 0 ffice 


State (that is, countrv) of nationality: 
SG 


State (that is. country) of residence: 

us 


This nerson is applicant l 1 all designated j 1 all desianated States except the United States | | the States indicated in 

for the purpoLs of 1 1 States ^ | | the United States of Ame^ca LiJ of America only | j the Supplcmcmal Box 


Name and address: (Family namefollowed by given name: for a legal entity, fidl official designation. 
The address must include postal code and name of country. The country of the address indicated in this 
Box is the applicant 's State (that is, country) of residence if no State of residence is indicated below.) 

LI, Dan 

Room 301, BIdg. 83 
PingJiYiCun. HongXin Road 
Shanghai 200336 China 


Til is person is: 

1 1 applicant only 

[ X| applicant and inventor 

1 1 inventor only (If this check-box 

1 1 is marked, do not fill in below.) 


Applicant's registration No. with the Office 


State (that is, country) of nationality: 

CN 


State (ihal is, coiinirv) of residence: 

CN 


This person is applicant j 1 all designated j 1 all designated States except v 

for the purposes of: 1 1 States I I the United States of Amenca \J1J 


the United States 1 1 the States indicated in 
of America only | | the Supplemental Box 



Name ana aaaress: (Family namejollowea Oy given name: jora legat enniy.juu ojjiciai ae:,ignuiiun. 
The address must include postal code and name ^country. Vie country of the address indicated in this 
Box is the applicant s State (that is, cotmtry) of residence if no State of residence is indicated below.) 

TANG, Xing 

Room 1201. BIdg. 39 

155 Lane of Lianhua (South) Road 

Shanghai 200237 China 



This person is: 

I I applicant only 

I X| applicant and inventor 

□ inventor only (If this check-box 
is marked, do not fill in below.) 



Apphcanfs registration No. with the Office 



State (that is, country) of nationality: 
CN 



State (that is, country) of residence: 

CN 



This person is aoDlicant l 1 all desienated | 1 all designated States except fyl the United States I | the States mdicated in 

for theTurSoLs o^^ LJ States | | the United States of Amenca IXJ of Amenca only | | the Supplemental Box 



I I Further applicants and/or (further) inventors are indicated on another continuation sheet. 



Form PCT/RO/101 (continuation sheet) (March 2001 ; reprint July 2003) 



See Notes to the request form 



Pr.T/CN 03/ 0 10 



Sheet No. ...3... ^ 

Box No. V DESIGNATION O F STATES Mark the appUcable check-boxes below; at least one must be marked 

The following designations are hereby made under Rule 4.9(a): 
Regional Patent 

18 AP ARIPO Patent: GH Ghana, GM Gambia, KE Kenya, LS Lesotho, iMW Malawi, MZ Mozambique, SD Sudan, 
SL Sierra Leone, SZ Swaziland, TZ United Republic of Tanzania, UG Uganda, ZM Zambia, ZW Zimbabwe, and any other 
Stale which is a Contracting State of the Harare Protocol and of the PCT (if other kind of protection or treatment desired, 
specify on dotted line) 

EA Eurasian Patent: AM Armenia, AZ Azerbaijan, BY Belarus, KG Kyrgyzstan, KZ Kazakhstan, MD Republic of Moldova, 
RU Russian Federation, TJ Tajikistan, TM Turkmenistan, and any other State which is a Contracting State of the Eurasian 
Patent Convention and of the PCT 

B EP European Patent: AT Austria, BE Belgium, BG Bulgaria, CH & LI Switzerland and Liechtenstein, CY Cyprus, CZ Czech 
Republic, DE Germany, DK Denmark, EE Estonia, ES Spain, FI Finland, FR France, GB United Kingdom, GR Greece, 
HU Hungary, IE Ireland, IT Italy, LU Luxembourg, MC Monaco, NL Netherlands, PT Portugal, RO Romania, SE Sweden, 
SI Slovenia, SK Slovakia, TR Turkey, and any other State which is a Contracting State of the European Patent Convention 
and of the PCT 

18 OA OAPI Patent: BF Burkina Faso, BJ Benin, CF Central African Republic, CG Congo, CI Cote d'lvoire, CM Cameroon, 
GA Gabon, GN Guinea, GQ Equatorial Guinea, GW Guinea-Bissau, ML Mali, MR Mauritania, NE Niger, SN Senegal, 
TD Chad, TG Togo, and any other State which is a member State of OAPI and a Contracting State of the PCT (if other kind 

of protection or treatment desired, specify on dotted line) 

National Patent (if other kind of protection or treatment desired, specify on dotted line): 

HR Croatia B OMOman 

HU Hungary Bi PG Papua New Guinea 

ID Indonesia SI PH Philippines 

IL Israel Q(l PL Poland 

IN India 53 PT Portugal 

IS Iceland B RO Romania 

JP Japan BI RU Russian Federation 

KE Kenya 

KG Kyrgyzstan SI SC Seychelles 

KP Democratic People's Republic 18 SD Sudan 

of Korea QB SE Sweden 

KR Republic of Korea 81 SG Singapore 

KZ Kazakhstan SI SK Slovakia 

LC Saint Lucia SI SL Sierra Leone 

LK Sri Lanka SB SY Syrian Arab Republic 

LR Liberia SI TJ Tajikistan 

LS Lesotho BI TM Turkmenistan 

LT Lithuania BI TN Tunisia 

LU Luxembourg BI TR Turkey 

LV Latvia Kl TT Trinidad and Tobago 

MA Morocco 

MD Republic of Moldova SI TZ United Republic of Tanzania 



81 


AT TIriitpH Arnh Pmirate^i 


KI 




AG Antigua and Barbuda 


KI 




AL Albania 


SB 


sa 


AM Armenia 


SB 


60 


AT Austria 


BB 


KI 


AU Australia 


BB 


B 


AZ Azerbaijan 


KI 


SI 


BA Bosnia and Herzegovina 

BB Barbados 


KI 

a 


SQ 


BG Bulgaria 


SI 


BI 


BR Brazil 




KI 


BY Belarus 


Bl 




BZ Belize 


BI 


BI 


CA Canada 


SI 


la 


CH 4S: LI Switzerland and Liechtenstein 


BO 




CN China 




SI 




SI 


sa 


CR Costa Rica 






CU Cuba 




Bl 


CZ Czech Republic 




SI 


DE Germany 


KI 


SI 






KI 


DM Dominica 




SI 


DZ Aleeria 


Bl 


SI 




Bl 


63 


EE Estonia 






ES Spain 


a 


SI 


FI Finland 


Kl 


SI 


GB United Kingdom 


a 
a 


B) 


GD Grenada 


BI 




KI 


BI 


GH Ghana 


KI 


a 


GM Gambia 


Bl 



KI UA Ukraine 

KI UG Uganda 

KI US United States of America . 



Macedonia 

MN Mongolia SB UZ Uzbekistan 

MWMalawi Kl VC Saint Vincent and the Grenadines 

MX Mexico B VN Viet Nam 

MZ Mozambique Kl YU Serbia and Montenegro 

NI Nicaragua ZA South Africa 

NO Norway Kl ZM Zambia 

NZ New Zealand Kl ZW Zimbabwe 

Check-boxes below reserved for designating States which have become party to the PCT after issuance of this sheet: 

□ □ □ 



Precautionary Designation Statement: In addition to the designations made above, the applicant also makes under Rule 4.9(b) all 
other designations which would be permitted under the PCT except any designation(s) indicated in the Supplemental Box as being 
excluded from the scope of this statement. The applicant declares that those additional designations are subject to confirmation and that 
any designation which is not confirmed before the expiration of 1 5 months from the priority date is to be regarded as withdrawn by the 
applicant at the expiration of that time limit. (Confirmation (including fees) must reach the receiving Office within the 15-month time limit.) 
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Box No. VI PRIORITY CLAIM 



The priority of the following earlier application(s) is hereby claimed: 



Filing date 
of earlier application 
(day/month/year) 



Number 
of earlier application 



Where earlier application is: 



national application: 
country or Member 
of WTO 



regional application:* 
regional Office 



international application: 
receiving Office 



item (1) 



item (2) 



item (3) 



item (4) 



item (5) 



I I Further priority claims are indicated in the Supplemental Box. 

The receiving Office is requested to prepare and transmit to the International Bureau a certified copy of the earlier application(s) (only 
if the earlier application was filed with the Office which for the purposes of this international application is the receiving Office) identified 
above as: 

□ all items DitemCl) Q item (2) □ item (3) . Q item (4) □ item (5) □ l^p^p^ie'^enia, Qox 

* Wliere the earlier application is an ARIPO application, indicate at least one countty party to the Paris Convention for the Protection of 
Industrial Property or one Member of the World Trade Organization for which that earlier application was filed (Rule 4.l0(b)(ii)): .... 

Box No. VTT INTERNATIONAL SEARCHING AUTHORITY 

Choice of International Searching Authority (ISA) (if two or more International Searching Authorities are competent to carry out the 
international search, indicate the Authority chosen; the nvo-letter code may be used): 

ISA/.C.N 

Request to use results of earlier search; reference to that search (if an earlier search has been carried out by or requested from the 
International Searching Authority): 

Date (day/month/year) Number Country (or regional Office) 



Box No. Vin DECLARATIONS 



The following declarations are contained in Boxes Nos. VIII (i) to (v) (mark the applicable 
check-boxes below and indicate in the right column the number of each type of declaration)'. 



Number of 
declarations 



□ Box No. Vin (i) 

□ Box No. VIII (ii) 

□ Box No. VIII (iii) 
Q Box No. VIII (iv) 

□ Box No- VIII (v) 



Declaration as to the identity of the inventor 

Declaration as to the applicant's entitlement, as at the international filing 
date, to apply for and be granted a patent 

Declaration as to the applicant's entitlement, as at the international filing 
date, to claim the priority of the earlier application 

Declaration of inventorship (only for the purposes of the designation of the 
United States of America) 

Declaration as to non-prejudicial disclosures or exceptions to lack of novelty 
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Box No. IX CHECK LIST; LANGUAGE OF FILING 



This international application contains: 

(a) in paper form, the following number of 

sheets: 

request (including 

declaration sheets) : 5 



description (excludins 
sequence listings and/or 
tables related thereto) 

claims 

abstract 

drawings 

Sub-total number of sheets 

sequence listings 

tables related thereto 

(for both, actual number of 
sheets if filed in paper form, 
whether or not also filed in 
computer readable form: 
see (c) below) 

Total number of sheets 



17 
4 
1 
9 



36 



36 



(b) □ only in computer readable form 

(Section 80l(a)(i)) 

(i) n sequence listings 

(ii) Q tables related thereto 

(c) Q also in computer readable form 

(Section 801(a)(ii)) 

(i) □ sequence listings 

(ii) □ tables related thereto 

Type and number of carriers (diskette, 
CD-ROM, CD-R or other) on which are 
contained the 

n sequence listings: 

□ tables related thereto: 

(additional copies to be indicated under 
items 9(ii) and/or lO(ii), in right column) 



This international application is accompanied by the following 
item(s) (mark the applicable check-boxes below and indicate in 
right column the number of each item): 



Number 
of items 



L 81 

2. □ 

3. □ 

4. □ 

5. □ 

6. □ 



10 



11 



. □ 
. □ 
. □ 

(i) 
(ii) 

(iii) 

(i) 
(ii) 

(iii) 
. □ 



fee calculation sheet 

original separate power of attorney 

original general power of attorney 

copy of general power of attorney; reference number, 

if any: : 

statement explaining lack of signature : 

priority document(s) identified in Box No. VI as 

item(s): * 

translation of international application into 

(language): : 

separate indications concerning deposited microorganism 

or other biological material : 

sequence listings in computer readable form 
(indicate type and number of carriers) 

□ copy submitted for the purposes of international search under 
Rule \jter only (and not as part of the international application) : 

□ (onlv where check-box (b)(i) or (c)(i) is marked in left column) 
additional copies including, where applicable, the copy for the 
purposes of international s^earch under Rule 1 3ter : 

□ together with relevant statement as to the identity of the copy or 
copies with the sequence listings mentioned in left column : 

tables in computer readable form related to sequence listings 
(indicate type and number of carriers) 

n copy submitted for the purposes of international search under 
Section S02{b-quater) only (and not as part of the international 
application) 

rn (only where check-box (b) (ii) or (c)(ii) is marked in left column) 
additional copies including, where applicable, the copy for the 
purposes of international search under Section Z02{b-quater) 

□ together with relevant statement as to the identity of the copy or 
copies with the tables mentioned in left column 

oihox (specify)'. 



Figure of the drawings which 
should accompany the abstract: 



Language of flling of the 

international application: 



English 



Box No. X SIGNATURE OF APPLICANT, AGENT OR COMMON REPRESENTATIVE 

Next to each signature, indicate the name of the person signing and the capacity in which the person signs (if such capacity is not obvious fivm reading the request). 





\. Date of actual receipt of the purported * r^rr (^ O t O H 0\ 
international application: 1 '3 U llo tU»JJ Vi ^7. I Z. ^ Df 


2. Drawings: 
1 1 received: 

1 1 not received: 


3. Corrected date of actual receipt due to later but 
timely received papers or drawings completing 
the purported international application: 


4. Date of timely receipt of the required 
corrections under PCT Article 1 1(2): 


5. International Searching Authority 

(if two or more are competent): ISA / 


6. 1 — 1 Transmittal of search copy delayed 
1 1 until search fee is paid 



For International Bureau use only 



Dale of receipt of the record copy 
by the International Bureau: 
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FEE CALCULATION SHEET 
Annex to the Request 



Applicant's or agent's 
file reference 



P3959/BST 



Applicant 

Intel Corporation, etc. 



International Application No 



PCT/CN 03/ 0 10 8 7 



f 9 0^'^ 2003 (1 9. 1 2. 0 3) 

Date stamp of the receiving Office 



CALCULATION OF PRESCRIBED FEES 

L TRANSMITTAL FEE | 

2. SEARCH FEE | 

Imcmaiional search to be carried out by 

(If two or more Iniemaiional Searching Authorities are competent to carry out the international 
search, indicate the name of the Authority which is chosen to carry out the international search.) 

3. INTERNATIONAL FEE 
Basic Fee 

Where items (b) and/or (c) of Box No. IX apply, enter Sub-total number of sheets 
Where items (b) and (c) of Box No. IX do not apply, enter Total number of sheets / 



CNY 500 



CNY1500 fs" 



36 



bl 



first 30 sheets 



CHF 650 IbTI 



b2 



15 



fee per sheet 



number of sheets 

J I in excess of 30 

tb3j additional component (only if sequence listinas and/or tables related 
thereto are filed in computer readable form under Section S01(a)(i), 
or both in that form and on paper, under Section 801(a)(ii)): 



CHF 90 



b2 



400 X 



= 1 



fee per sheet 

Add amounts entered at b I , b2 and b3 and enter total at B . . 
Designation Fees 

The international application contains designations. 



CHF 740 fil 



140 



number of designation fees 
payable (maximum 5) 



= L 



amount of designation fee 



CHF 700 [dI 



Add amounts entered at B and D and enter total at I 
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Electronic Pen-Computer Multimedia Interactive System 

BACKGROUND 

1. FIELD 

The present invention relates generally to connputer systems and, more 
specifically, to new usage models for personal computers (PCs) in a multimedia 
interactive mode. 

2. DESCRIPTION 

Many different input devices have been developed over the years for 
computer systems such as PCs. Devices such as mice, trackballs, touch 
screens, joysticks, and so on have been used by both sophisticated and 
unsophisticated computer users to varying degrees of success. For some 
unsophisticated users such as small children or the elderly, manipulating these 
input devices has sometimes been problematic, thereby potentially limiting their 
enjoyment in using PCs. 

One relatively recent advance in input methods is the electronic pen. 
Users may use an electronic pen to write on traditional paper. The electronic 
pen sends a wireless signal containing coordinate information to a receiver. 
The receiver can calculate the pen tip's position information and send this 
information to the PC. This solution is one of the most natural input methods 
for a PC. The user can write anything on normal paper. The electronic pen 
can obtain the absolute position of the pen tip without any dependence on the 
contents of the written material. Some applications have already been enabled 
using electronic pen technology. In one example, a pen application can 
capture a user's handwriting and integrate what has been written into an 
application program (such as an electronic mail (e-mail) program) in order to 
send an e-mail containing an image of the handwriting. The electronic pen has 
also been used as a selection device (much as a mouse is used) to control a 
window's cursor. 
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Because use of the electronic pen is more natural than other input 
methods, it may find utility, especially for unsophisticated computer users. For 
example, small children and the elderly who have difficulty using a keyboard 
may be able to effectively input information using the pen. The electronic pen 
may be especially useful for entering information in Asian languages (such as 
Chinese) into computers. 

In addition, electronic books and/or courseware have become more 
popular because of the inherent support in these materials for multimedia 
functions. Users may enjoy audio, video, and still images on a PC and get 
more information from these materials than from a text-only product. However, 
readers continue to enjoy the convenience and widespread availability of 
conventional paper books. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The features and advantages of the present invention will become 
apparent from the following detailed description of the present invention in 
which: 

Figure 1 is a diagram of an electronic pen-computer multimedia 
interactive system according to an embodiment of the present invention; 

Figure 2 is a diagram illustrating an electronic pen system according to 
an embodiment of the present invention; 

Figure 3 is a flow diagram of a maker component and a player 
component in a multimedia system according to an embodiment of the present 
invention; 

Figure 4 is a flow diagram of player processing according to an 
embodiment of the present invention; 

Figure 5 is a flow diagram of maker processing according to an 
embodiment of the present invention; 
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Figure 6 is an example of object identification processing according to 
an embodiment of the present invention; 

Figure 7 is a flow diagram of action linkage processing according to an 
embodiment of the present invention; 

Figure 8 is an example of a page in a book supported by the present 
system; and 

Figure 9 is a flow diagram of a logic behavior linking process according 
to an embodiment of the present invention. 



DETAILED DESCRIPTION 

Embodiments of the present invention introduce a new usage model for 
computer systems based on the combination of an electronic pen position 
location system and traditional paper books. Embodiments of the present 
invention provide a method to link electronic books and/or courseware with 
traditional paper books. By using the present invention, a user can enjoy 
multimedia content on a computer system (such as a PC), while reading a book. 
The present invention provides interactivity offered by a PC, but based directly 
on content selected from an existing paper book. Portions of the book's 
contents may be processed by the system prior to usage by a user in order to 
set up the linkages between the book's contents and multimedia content stored 
on the computer system. 

Reference in the specification to "one embodiment" or "an embodiment" 
of the present invention means that a particular feature, structure or 
characteristic described in connection with the embodiment is included in at 
least one embodiment of the present invention. Thus, the appearances of the 
phrase "in one embodiment" appearing in various places throughout the 
specification are not necessarily all referring to the same embodiment. 

Figure 1 is a diagram of an electronic pen-computer multimedia 
interactive system 100 according to an embodiment of the present invention. In 
embodiments of the present invention, a computer system 102 interacts with a 

3 



W/CN03/OiO- 8 7 

traditional paper book 104 being read by a user. The term "book" as used 
herein may be any pre-existing printed material, such as a fiction or non-fiction 
book, a periodical such as a magazine, newspaper, or a joumal. a coloring 
book, a pamphlet, an advertisement, and so on. The book may be placed by 
the user so as to reside on a book holder 106, which holds the book in a fixed 
relationship to a receiver device 108. An electronic pen (E-pen) 110 may be 
used by the user to point to and/or write on the book 104. The position of the 
electronic pen may be determined by the receiver and forwarded over a 
communications link 112 to the computer system. The communications link 
may comprise any wired or wireless connection, such as a COM port, universal 
serial bus (USB), LPT, any variation of implementation of the 802.11 wireless 
standards, Bluetooth, infrared, and so on. 

The computer system includes at least one database and an application 
1 14 to accept the position information of the electronic pen and an indication of 
what book is being read by the user, and to render multimedia content on the 
computer system or to provide a selected action in response to reception of the 
position information. Generally, the responses may include playing one or 
more of audio content, playing video content, displaying an image or text, or 
any other multimedia information, or taking a pre-defined action on the 
computer system (such as adjusting the speaker volume, tuning the display 
brightness, or powering off the computer system, for example). The responses 
may be set up in advance of the user reading the book such that the application 
executing on the computer system links position information of the electronic 
pen over portions of pages of the conventional, pre-existing published book 
with selection of an appropriate computer system response. 

Embodiments of the present invention may be used as follows. The 
receiver 108 may be fixed on top of the book holder 106, while the book 104 is 
put on the book holder by the user. This means the distance between the book 
and receiver is fixed. In at least one embodiment, the fixed relationship may 
include one corner and two sides of the book being situated against a fixed 90 
degree angle structure of the book holder. When the user uses the electronic 
pen 110 to select contents of the book on a currently viewed page, the tip 
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position information may be calculated by the receiver 108 and sent to the 
computer system 102. At least one database may be included in the computer 
system. The at least one database contains action/response information 
corresponding to the current E-pen position, the contents of the book for the 
current page, and stored multimedia contents. The database or portions 
thereof may be local to the computer system 102, or may be stored and 
accessed externally from the computer system using well-known techniques, 
such as over the Internet, for example. An application receives the position 
information, searches the at least one database for an entry correlated to the 
position information and book contents, and then plays the con-esponding 
multimedia content or takes a corresponding pre-defined action. In at least one 
embodiment, the multimedia content may be played on consumer electronics 
devices such as televisions, stereos, computer display monitors, etc., that may 
be communicatively coupled with the computer system. 

One usage of embodiments of the present invention is the teaching of 
the English alphabet. For example, the book may contain 26 pages, each page 
containing a letter of the alphabet. A first page may contain a large letter "A" 
and pictures of different types of apples or other objects starting with the letter 
"A". When the user selects the letter "A" on that page of the book (perhaps by 
depressing a button on the E-pen or touching the E-pen to the paper), the 
application on the computer system causes the computer to emit the sounds for 
the word "apple", for example, via the computer system's speaker. At 
substantially the same time, the computer system may aiso be used to show 
more pictures or play a video of apples of different sizes and colors (for 
example). Thus, embodiments of the present invention take the contents of a 
pre-existing book and combine this content with the interactivity of a computer 
system for enhanced information flow and education/entertainment purposes. 
When used for education, the present invention combines the power of an 
existing or newly published traditional paper book with the power of a computer 
system (in terms of multimedia, and Internet access in some embodiments). 

With embodiments of the present invention, many other usage modes 
are possible. For example, different modes of educational systems may be 
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designed for different age groups. For children two to four years of age, an 
application and related database that teach simple words, alphabets, and 
numbers can be created based on existing books for such purposes. For 
children five to seven years of age, an application and related database that 
.teach sentences can be created. In other settings, other modes are possible, 
and the present invention is not limited in this respect. For example, an 
application and related database that allows tourists to get more information 
from an existing printed map can be created. In another example, an 
application that allows for virtual remote control for electronic devices in the 
home may be created using the system and techniques described herein. 

Figure 2 is a diagram illustrating an electronic pen system according to 
an embodiment of the present invention. The E-pen system obtains absolute 
tip position information on a page of a book. At least two ultrasound sensors 
SI 200 and S2 202 may be arranged on a line in the receiver 108, equidistant 
from a coordinate system origin (0, 0), as noted by (D, 0) and (-D, 0), 
respectively. The E-pen 110 comprises an ultrasound transmitter T 204. When 
the pen tip is down, the transmitter sends an ultrasound signal detectable by 
the receivers. Since the distances LI = (T, S1) and L2 = (T, S2) are different, 
the ultrasound signal reaches the two sensors at different times. The receiver 
108 may detect the time difference and calculate the length of LI and L2. The 
position of T may then be calculated by equation 1 : 



The X and Y coordinates specify the absolute position information of the 
pen tip. The E-pen system sends this information to the computer system 1 02 
over connection 112. 

Once the position information is received by the user's computer system 
102, the information may be processed by an application program called a 
"player" component herein. The player receives the position information, 
searches in at least one database for corresponding multimedia content and/or 
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an appropriate action, and plays the corresponding nnultimedia and/or performs 
the action. The user interacts with the player in order to make a traditional 
paper book into an interactive user experience. The user uses the E-pen to 
select an object on the book, and the player provides the corresponding 
response based on the pen tip position and the object type. 

Prior to use, at least one database must be generated to contain 
action/response information relating the pen tip position, current book contents, 
current page of the book, multimedia contents for the current book, and pre- 
defined actions for the current book. An application program executing on a 
publisher's computer system called a "maker" component herein may be used 
to create the linkages between these data items. The maker may be used by a 
publisher (or in some cases by a sophisticated user) to make an interactive 
multimedia product based on an existing traditional paper book (or possibly a 
newly published book), with limited manual work. The input data to the maker 
includes the page images of the book, additional multimedia content, and pre- 
defined actions. The publisher selects an object on a page image, and liriks 
the relevant multimedia content and/or action with the selected object. All 
linkage information may be stored in the at least one database. 

In one embodiment, the publisher's computer system may be the same 
as the user's computer system for the case where the interactive multimedia 
product and/or the book are produced by the user or another person at the 
user's site (e.g., a parent prepares the linkages for use by a child). In one 
embodiment, the multimedia content may comprise one or more of digital 
images, digital video, digital audio files, and text files produced by or obtained 
by the user. In one embodiment, the user may provide digital images as the 
pages of the book (e.g., the user's original digital photographs) and specify the 
linkages of objects on each page of the book with the multimedia content. 

Figure 3 is a flow diagram of a maker component 300 and a player 
component 302 in a multimedia system according to an embodiment of the 
present invention. At block 304. the maker component 300 accepts as input 
images of one or more pages of a book and then defines zone information for 
objects represented on those pages. In one embodiment, the maker may 
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accomplish this through interaction with a system user acting as a publisher of 
the multimedia content. At block 308, the maker sets up the linkage between 
pen positions, the contents of the book (e.g., objects found on pages of the 
book), and related multimedia contents (e.g., associated audio, video, images, 
text, and so on) and/or actions, using multimedia files and actions 310 as input 
data. At block 312, the maker packages the databases for subsequent use by 
the player component 302. In one embodiment, the publisher provides the 
multimedia files when operating the maker component and installs them in the 
databases distributed to the user's computer system with the player component. 
In another embodiment, the publisher provides links to the multimedia files 
accessible over a network such as the Internet. In this embodiment, the player 
component obtains the files as needed using a known file transfer protocol or 
the well known hyper text transfer protocol (HTTP). 

The databases comprise at least a book content database 314, a 
multimedia database 316, and an action library 318. The book content 
database contains, the images of pages of the book, positional information 
about objects on the pages, and linkage information between the objects and 
multimedia contents or actions/responses. The multimedia database contains 
audio, video, images, text, and other data corresponding to objects on pages of 
the book for rendering for perception by the user. The action library 318 
contains directives for pre-defined actions to be performed by the user's 
computer system when the position of the electronic pen coincides with an 
object on a page of the book indicating the associated action. In one 
embodiment, the actions may include selecting a human language for 
subsequent communication, adjusting the volume of a speaker, adjusting the 
brightness of a computer monitor, sending commands to an input/output (1/0) 
port, powering off the computer system, and so on. Other actions are also 
contemplated. In one embodiment, the three databases described herein may 
be combined into a single database. 

The databases corresponding to the book may then be distributed using 
any known mechanism to the user's computer system. After the player 
application is installed on the user's computer system, the user may put the 
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book on the book holder, start up the player component 302 on the computer 
system, and begin to interact with the book. When the user selects an area on 
the book or book holder that has meaning to the system, the user has selected 
a valid input. In one embodiment, selecting other areas by the user may be 
ignored by the system. At block 320, the user selects an object represented on 
a page of the book using the electronic pen. At block 322, the receiver 
calculates the current pen tip position during the selection. At block 324, the 
receiver 108 sends the pen tip position information to the computer system 102 
using the connection 1 12. At block 326, the player component mnning on the 
computer system receives the position information and searches the book 
content database 314 to obtain information relating to an object (if any) 
selected by the current pen position on the current page. If an object is 
selected and multimedia content and/or an action are associated with the 
selected object, the player component plays the corresponding multimedia 
content and/or performs the specified action at block 328. The player 
component uses data from the multimedia database 316 and the action library 
318 in performing block 328. The player component directs the playing of the 
content on the computer system and/or a communicatively coupled consumer 
electronics device such as a television. 

Figure 4 is a flow diagram of player processing according to an 
embodiment of the present invention. The player receives the current position 
information from the electronic pen system at block 400. At block 402, the 
player detemiines, in one embodiment, based on the current position 
information of the pen, if the user is selecting a special area on the page of the 
book, or on the receiver or book holder. If so, the player processes, in one 
embodiment, a selected pre-defined action at block 404, using the action library 
318 as input data. Pre-defined actions may include such things as increasing 
or decreasing the volume of a speaker, adjusting visual attributes of a computer 
monitor, selecting a human language for subsequent communication, and so 
on. Other actions may be defined for specific implementations and the present 
invention is not limited in this respect. In one embodiment, the action may be 
used to control the player component or the computer system, these actions 
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having no relationship with the book's contents. If the current pen position 
information does not indicate a special area, then the player compares the 
position information with known objects for the page of the book as stored in 
the book content database 314. The player determines which object of the 
page is selected (if any) at block 406, based on boundary coordinates of the 
object and the current pen position information. If an object is selected, the 
player at block 408 gets the selected object's linkage information from one or 
more of the book content database, the multimedia database 316, and the 
action library 318. At block 410, the player plays the linked multimedia content 
or processes an action based on the object linkage information. 

Before the user can cause the rendering of related multimedia content or 
cause related actions to be performed on the computer system as a result of 
selecting an object on a page of a book using the electronic pen, the databases 
must be populated and the linkages between objects, positions and content 
and actions must be set up. This processing may be performed by a maker 
component according to embodiments of the present invention. The present 
maker solves the problem of how to create electronic material from a traditional 
book with limited effort by a publisher or user. Embodiments of the present 
invention provide a general method for creating actions/responses on a 
computer system from paper books. Based on the present invention, the 
publisher or user can create interactive multimedia material linked to the 
existing or newly published traditional paper book with limited manual work. 

The maker creates different actions or responses to selection of an 
object defined on a page of a book. Here, the term "object" may be any zone 
containing specific meaning on the page (for example, a boy's picture, a picture 
of an apple, and so on). With limited input by the publisher or user, the maker 
identifies the object on the page. In various embodiments, the responses 
include playing audio, video, showing an image, or rendering any other 
multimedia information, performing a pre-defined action, or executing a logic 
script based on input from the user. The input data for the maker include page 
images of a book 500. multimedia contents 502, and a pre-defined action 
library 318. The output data of the maker includes the book content database 
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314, the multimedia database 316. and the action library 318. The book 
content database contains the linkage relationship among objects on each 
page, multimedia contents in the multimedia database, and pre-defined actions 
from the action library. 

The maker provides three services. First, the maker identifies the 
objects on a page. The maker determines each object's boundary information 
based on limited user inputs. Second, the maker creates the linkage among 
the objects, actions, and responses. For example, if an audio file is linked to an 
object, when the user selects that object with the electronic pen, the player can 
then play that audio file. More than one multimedia file and one or more 
action/response may be linked to each object. Third, the maker packages the 
databases. When the publisher or user finishes setting up linkages for the 
pages of a book, the maker packages all of the information into the book 
content database, the multimedia database, and the action library for 
distribution with the player to a user's computer system. 

Figure 5 is a flow diagram of maker processing according to an 
embodiment of the present invention. At block 504, the maker creates new 
blank book databases to hold the book contents (e.g., object definitions and 
linkages), multimedia, and actions. At block 506, in one embodiment, the 
maker allows the publisher or user to identify an object zone by using the 
electronic pen or a mouse and image processing algorithms. The page images 
used by this operation may be obtained by scanning using a scanner or may be 
pre-existing image files. In one embodiment, a book may be put on the book 
holder and the page image may be displayed on a monitor of the computer 
system. In this scenario, the boundary of an object on the displayed page 
image may be used to define the object. 

In various embodiments, the maker may use one or more of several 
methods to identify the objects on the page image. In one embodiment, the 
publisher or user uses the electronic pen to draw a cun/e along the boundary of 
the object on the page of the book, or selects key points on that boundary. The 
electronic pen system sends the pen position information to the maker, and the 
maker thus obtains an array of position information of the object. Depending 
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on the publisher's drawing speed, the maker may rescale the set of points and 
mark them as an object boundary. Alternatively, the publisher may use a 
mouse to draw a curve along the boundary of the object on the page image as 
the page image is displayed on the monitor, or the publisher may modify a 
shape (such as a triangle, rectangle, ellipse or polygon) as is well known in the 
art to encapsulate the boundary of the object. In yet another embodiment, the 
publisher may use a mouse to select a set of boundary points of the object 
displayed on monitor. The maker then applies one or more known digital 
image processing algorithms to identify the object boundary automatically. For 
example, known "intelligent scissor** or "magic wand" algorithms may be used. 
In some embodiments, the algorithms identify objects by examining pixePs gray 
scale values, colors, and other image information. Figure 6 is an example of an 
identification of objects process according to one embodiment of the present- 
invention. In this example, the maker has identified several objects in the page 
image and created a boundary for each object. In this way, all objects may be 
determined for a page image. 

Next, at block 508, the publisher or user uses the maker to manually 
create a linkage among the objects, actions, and responses, the publisher or 
user selects an object and assigns actions and/or responses to the object. In 
at least one embodiment, there are three kinds of actions/responses. 

First, the player may play multimedia contents, including playing video, 
playing audio, showing one or more images, or showing text. In this case, the 
publisher links the multimedia files to an object. The maker logs this linkage 
information and saves the link to the book content database. 

Second, the player may execute a pre-defined action other than playing 
video, audio, images, or text. For example, the pre-defined action may be 
adjusting the volume of the speaker of the computer system, adjusting 
brightness of the monitor, sending commands to any I/O port, or any other 
action affecting the computer system. The pre-defined actions may be 
classified into groups. Each group may comprise a dynamic link library (DLL) 
510. A software interface may be defined between the DLLs and the maker. 
For example, in one embodiment, "GetActionList (LPSAction a_SActionArray, 
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DWORD dwArrayLength.)" and "CallActionBylD (INT ActionID)" application 
program interfaces (APIs) may be implemented for each DLL. The 
"GetActionList" function returns all valid action names and their action IDs. The 
maker can call this API and display the action name list on the maker 
component's user interface (Ul). When the publisher or user links an action 
name to an object, the maker assigns the corresponding Action ID to that 
object and saves this linkage information in the book content database. When 
the player wants to execute the action of the object, the player gets the action 
ID from the database and calls the "CallAction BylD (INT ActionID)" function. 
The CallActionBylDO invokes the specific action based on the Action ID. 

This processing is shown in Figure 7. Figure 7 is a flow diagram of 
action linkage processing according to an embodiment of the present invention. 
At block 700, the maker 300 calls the get action list API, which accesses the 
action list 702. Each entry in the action list may be indicated by a unique action 
ID. Some examples of actions are shown in Figure 7, but other actions may 
also be used. At block 704, the maker links the action ID to the object currently 
being processed. At block 706, the maker packages the linkage information to 
the action library 318. The action library may be subsequently read by the 
player component 302. At block 708, when a particular object is selected and 
the object is linked to an action, the player gets the action ID from the action 
library. At block 710, the player component calls the action by using the action 
ID and the CallActionBylD() API. The action is then performed by the computer 
system at block 712. 

Third, the player may execute complex logic behavior based on previous 
inputs from users. For example, suppose there are images of animals on the 
current page of the book, along with two buttons as shown in Figure 8. When a 
user selects the "English" button 800 first (using the electronic pen), and then 
selects any animal's picture on the page, the player plays the audio file (for 
example) which contains the name of the animal in the English language. 
However, when the user selects the "Chinese" button 802 first, and then selects 
an animal's picture, the player plays the audio file which contains the name of 
the animal in the Chinese language. This is merely an example of complex 
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logic behavior. IVIany other responses may be made based on successive user 
inputs. 

This logic behavior extends the behavior of an object by adding a 
publisher's own functionality. In one embodiment, it may be implemented by 
adding a script language extension ability to the mal<er 300. In various 
• embodiments, the script language may be any l<nown language for such a 
purpose, such as VBScript, JScript. JavaScript. PerlScript. PythonScript, etc. 
In one embodiment, there may be a script module for each page of the book. 
The publisher or user may design a script function for an object. A page global 
variable or other functions may also be defined in the script module. The 
maker may save the script module for each page in the action library 318. The 
player loads the script module for the current page from the action library and 
adds the script module to a corresponding script engine (e.g.. an executable 
module for processing scripts). When a user select an object With a script 
function assigned to it, the player will execute the script function using the script 
engine. 

Figure 9 is a flow diagram of this logic behavior linking process 
according to an embodiment of the present invention. At block 900. the maker 
defines the page global variable and functions for a selected page of the book. 
A simplified example of play audio and video functions is show at block 902. At 
block 904. the maker defines the script function for a selected object. A 
fragment of a script function example is shown at block 906. At block 908, the 
maker packages the linkage information and the script function as a script 
module and stores the linkage information and the script function in the action 
library. Subsequently, when the user is interacting with a current page of the 
book, the player loads the script module for the cun-ent page from the action 
library. At block 912. the player adds the script module to the script engine. In 
one embodiment, the script engine may be VBScript or Jscript as provided by 
versions of the Windows operating system commercially available from 
Microsoft Corporation. At block 916. the player may call the script function 
when the user selects an object linked to a script. The call to the script function 
results in the script engine processing the script associated with the selected 
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object. At block 918, the action is performed as a result of the operation of the 
script engine. 

Returning back to Figure 5, at block 512. the nnaker stores all linkage 
information, multimedia files, and other actions/responses into the appropriate 
places in the multimedia database, the book content database, and the action 
library. At block 514, in one embodiment, the maker generates a preview of the 
page in the book for verification by the publisher or user that the intended 
linkages have been stored. At block 516, the maker packages the databases 
518 for subsequent distribution to the user's computer system. 

Embodiments of the present invention provide at least the following 
advantages over existing multimedia systems. The user can use the electronic 
pen as the form of input. This is the most natural input method for a computer 
system such as a PC. With this input method, the present invention can be 
used by people who are not familiar with typing on a keyboard. The user can 
still read or look at the traditional book. In the meantime, the user can also 
control getting multimedia information from the PC. This means that the user 
can enjoy both the advantages of paper books and the multimedia capabilities 
that the computer offers, including audio, video, images, and text available 
locally or via a network such as the Internet, Additionally, new usage models 
are enabled by embodiments of the present invention. In just two examples, an - 
educational system can be provided using embodiments of the present 
invention which allows children to learn a language, and tourists can use the 
invention to get more active information from a traditional map. Many other 
usage models are possible with the present invention. 

Thus, described herein are methods and apparatus to accelerate the 
conversion from traditional books to interactive multimedia books requiring 
limited user input. A publisher may use the techniques described herein to 
build interactive materials based on existing paper books. The present 
invention provides multiple methods to identify objects on the page. A 
publisher or user can either use the electronic pen to define the boundary of the 
object on the paper, or use a mouse to define the boundary of the object on the 
computer system's monitor with the assistance of known digital image 
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processing algorithms. Additionally, actions may be enabled for a book. This 
means the user may control the computer system through interaction with a 
page of a book. Finally, logic behavior may be enabled for a book. This 
feature extends the book's contents in an infinite number of ways. For example, 
an existing book may be transformed Into a virtual multi-language learning book.' ' 
a virtual musical instrument, or a virtual remote controller for a consumer' 
electronics device. Other applications are contemplated with embodiments of 
the present invention. 

Although the operations described herein may be described as a 
sequential process, some of the operations may in fact be performed in parallel 
or concurrently. In addition, in some embodiments the order of the operations 
may be rearranged without departing from the spirit of the invention. 

The techniques described herein are not limited to any particular 
hardware or software configuration; they may find applicability in any computing 
or processing environment. The techniques may be implemented in hardware, 
software, or a combination of the two. The techniques may be Implemented in 
programs executing on programmable machines such as mobile or stationary 
computers, personal digital assistants, set top boxes, cellular telephones and 
pagers, and other electronic devices, that each include a processor, a storage 
medium readable by the processor (including volatile and non-volatile memory 
and/or storage elements), at least one input device, and one or more output 
devices. Program code is applied to the data entered using the input device to 
perform the functions described and to generate output information. The output 
information may be applied to one or more output devices. One of ordinary skill 
in the art may appreciate that the invention can be practiced with various 
computer system configurations, including multiprocessor systems, 
minicomputers, mainframe computers, and the like. The invention can also be 
practiced in distributed computing environments where tasks may be performed 
by remote processing devices that are linked through a communications 
network. 

Each program may be implemented in a high level procedural or object 
oriented programming language to communicate with a processing system. 
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However, programs may be implemented in assembly or machine language, if 
desired. In any case, the language may be compiled or interpreted. 

Program instructions may be used to cause a general-purpose or 
special-purpose processing system that is programmed with the instructions to 
perform the operations described herein. Alternatively, the operations may be 
performed by specific hardware components that contain hardwired logic for 
performing the operations, or by any combination of programmed computer 
components and custom hardware components. The methods described 
herein may be provided as a computer program product that may include a 
machine readable medium having stored thereon instructions that may be used 
to program a processing system or other electronic device to perform the 
methods. The term "machine readable medium" used herein shall include any 
medium that is capable of storing or encoding a sequence of instructions for 
execution by the machine and that cause the machine to perform any one of 
the methods described herein. The term "machine readable medium" shall 
accordingly include, but not be limited to, solid-state memories, optical and 
magnetic disks, and a carrier wave that encodes a data signal. Furthermore, it 
is common in the art to speak of software, in one form or another (e.g., program, 
procedure, process, application, module, logic, and so on) as taking an action 
or causing a result. Such expressions are merely a shorthand way of stating 
the execution of the software by a processing system cause the processor to 
perform an action of produce a result. 

While this invention has been described with reference to illustrative 
embodiments, this description is not intended to be construed in a limiting 
sense. Various modifications of the illustrative embodiments, as well as other 
embodiments of the invention, which are apparent to persons skilled in the art 
to which the invention pertains are deemed to lie within the spirit and scope of 
the invention. 
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CLAIMS 



What is claimed is: 

1 . A method of associating a selected object on any printed material 
to a valid response provided by a computer system comprising: 
determining a position on a page of the printed material; 
transmitting the position to the computer system; 
con-elating the position to selected content associated with the printed 
materials, the selected content being accessible by the computer system; and 

providing a valid response to a user based at least in part on the position 
and the correlated content, wherein the valid response includes at least one of 
rendering audio content, rendering video content, rendering image content, 
rendering text content, and performing an action by the computer system. 

2. The method of claim 1 , wherein determining the position comprises 
detecting ultrasound signal timing information generated by an electronic pen 
situated near the printed material. 



3. The method of claim 1, wherein correlating the position comprises 
searching a printed material database, the printed material database 
comprising positional information of objects on the pages. 

4. The method of claim 1 , wherein performing the action comprises 
selecting a language to be used for subsequent responses. 

5. The method of claim 1 , wherein performing the action comprises at 
least one of adjusting volume of a speaker, adjusting brightness of a computer 
monitor, sending commands to an input/output (I/O) port, and powering off the 
computer system. 
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6. The method of claim 1 , wherein the printed material comprises a 
traditional paper book. 

7. An article comprising: a storage medium having a plurality of machine 
accessible instructions, wherein when the instructions are executed by a 
processor, the instructions provide for associating a selected object on any 
printed material to a valid response provided by a computer system by 

determining a position on a page of the printed material; 

transmitting the position to the computer system; 

correlating the position to selected content associated with the printed 
materials, the selected content being accessible by the computer system; and 

providing a valid response to a user based at least in part on the position 
and the correlated content, wherein the valid response includes at least one of 
rendering audio content, rendering video content, rendering image content, 
rendering text content, and performing an action by the computer system. 

8. The article of claim 7, wherein instructions for determining the 
position comprise instructions for detecting ultrasound signal timing information 
generated by an electronic pen situated near the printed material. 

9. The article of claim 7, wherein instructions for correlating the position 
comprise instructions for searching a printed material database, the printed 
material database comprising positional information of objects on the pages. 

10. The article of claim 7, wherein instructions for performing the action 
comprise instructions for selecting a language to be used for subsequent 
responses. 

1 1 . The article of claim 7, wherein the action comprises at least one of 
adjusting volume of a speaker, adjusting brightness of a computer monitor, 
sending commands to an input/output (I/O) port, and powering off the computer 
system. 
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12. A system for associating a selected object on any printed material to 
a valid response provided by a computer system comprising: 

a pointing device to determine a position on the printed material; 

a communicating device to. transmit the position to the computer system; 

a player component to correlate the position to selected content 
associated with the printed materials, the selected content being accessible by 
the computer system; and to provide a valid response to a user based at least 
in part on the position and the correlated content, wherein the valid response 
includes at least one of rendering audio content, rendering video content, 
rendering image content, rendering text content, and performing an action by 
the computer system. 

13. The system of claim 12, wherein the pointing device comprises an 
electronic pen. 

14. The system of claim 12, further comprising a holder structure to hold 
the printed materials in a fixed relationship to the pointing device. 

15. The system of claim 12, further comprising a multimedia database 
to store digital multimedia content, a printed material content database to store 
positional information about objects on the pages and linkage information 
between the objects and at least one of the multimedia contents and actions, 
and an action library to store directives for actions to be performed on the 
system. 

16. The system of claim 12, wherein the printed material comprises a 
traditional paper book. 

17. The system of claim 12, wherein the action comprises selecting a 
language to be used for subsequent responses. 
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18. The system of claim 12, wherein the action comprises at least one 
of adjusting volume of a speaker, adjusting brightness of a computer monitor, 
sending commands to an input/output (I/O) port, and powering off the computer 
system. 
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ABSTRACT OF THE DISCLOSURE 

A system for associating a selected object on any printed material to a 
valid response provided by a computer systiem includes a pointing device to 
determine a position on the printed material, a communicating device to 
transmit the position to the computer system, and a player component to 
correlate the position to selected digital content associated with the printed 
materials, the selected digital content being accessible by the computer system; 
and to provide a valid response to a user based at least in part on the position 
and the correlated content, wherein the valid response includes at least one of 
rendering audio content, rendering video content, rendering image content, and 
performing an action by the computer system. In one embodiment, the pointing 
device is an electronic pen and the printed material is a traditional paper book. 



22 




i 



PCT/CN 03/ C iO 8 7 




1/9 




Pa/CN 03/ 0 10 8 7 



7 




T (X. Y) 



ULTRASOUND 
TRANSMITTER 



Figure 2 



Pi:T/CN 03/ 0 10 8 : 



i ■ 



3/9 



300 



320 



USER SELECTS OBJECT ON 
A PAGE OF THE BOOK USING 
THE ELECTRONIC PEN 



"32r 

/ 



RECEIVER CALCULAT ES 
PEN TIP POSITION 

J 



RECEIVER SENDS POSIT ION 
INFORMATION TO COMPUTER 
SYSTEM THROUGH THE 
CONNECTION 



GET THE POSITION 
INFORMATION AND SEARCH 
THE BOOK CONTENT DATABASE 



/ 
326 



328 



PLAY THE MULTIMEDIA 
CONTENT AND/OR PERFORM 
SPECIFIED ACTION 



306 



IMAGES OF 

BOOK 
CONTENTS 



MAKER 





304 

r / 


DEFINE OBJECTS 
ON PAGES OF THE BOOK 



303 

A 



SET UP THE LINKAGE 
AMONG PEN POSITIONS. CONTENTS 
OF THE BOCK AND RELAT ED 
MULTIMEDIA CONTENTS 



312 



PACKAGE THE 
DATABASES 



L-L^"-===:==-=:= 



310 

_A 



/IviULTIMEDIA FILEs'N [ 
l,^ AND ACTIONS J I 



BOOK 
CONTENT 
DATABASE 



PLAYER 



302 



i14 



MULTIMEDIA 
DATABASE 



318^ 



ACTION 
LIBRARY 



316 



Figure 3 



PCT/CN 03/ 0 10 8 7 



4/9 



400 

/ 

RECEIVE POSITION INFORMATION 
FROM THE ELECTRONIC PEN SYSTEM 




404 

/ 



YES 



PROCESS 
PRE-DEFINED 
ACTION 





.NO • 


COMPARE POSITION WITH 
INFORMATION IN BOOK 
CONTENT DATABASE. DETERMINE 
WHICH OBJECT IS SELECTED 







314 



BOOK 
CONTENT 
DATABASE 



408 

/ 



GET SELECTED OBJECT 
LINKAGE INFORMATION 
FROM DATABASES 




f MULTIMEDIA^ 
\^ DATABASE J 



PLAY MULTIMEDIA CONTENT OR 
PROCESS EVENT ACTION . 




Figure 4 



^T/CN {iV 0 10 8 i 



"A/ 



5/9 



504 
/ 



CREATE NEW BLANK BOOK 
DATABASES 



506 



IDENTIFY THE OBJECT ZONE 
BY USING THE E-PEN OR MOUSE 
AND IMAGE PROCESSING 
ALGORITHMS 



500 
/ 



PAGE IMAGES OF 
PAPER BOOK 
CONTENT 



510 

^ ^ 

ACTION 

DLL 



503 



502 



LINK THE ACTIONS/RESPONSES 
WITH THE OBJECT 



MULTIMEDIA FILES 
AND OTHER 
RESPONSES 



512 
/ 



STORE ALL LINKAGE INFORMATION, 
MULTIMEDIA FILES AND OTHER 
ACTIONS/RESPONSES INTO 
DATABASES 



314 



514 
/ 



GENERATE A PREVIEW 
OF THE PAGE IN THE BOOK 



516 



PACKAGE THE 
DATABASES 



MULTIMEDIA 
DATABASE 



A 
316^ 



BOOK 
CONTENT 
DATABASE 



ACTION 
LIBRARY 



318 



Figure 5 




Figure 6 



PCT/CN 03/ 0 iO 8 7 



7/9 



5;o 



300 

, J. 

MAKER 



_ irr: — i 



702 

/ 



ACTION 
DLL 



700 



GET ACTION LIST () 



704 



LINK THE ACTION ID 
TO THE OBJECT 



706 

jL 



ACTION 1: SPK VOLUME UP 
ACTION 2: SPK VOLUME DOWN 
ACTION 3: SPK MUTE 



PACKAGE THE LINKAGE 

INFORMATION TO THE 
ACTION LIBRARY 



313 



OBJ 1 ACTION 3 
OBJ 2 -> ACTION 1 



510 
^ 

'action ^ 

DLL 



r 



302 



PLAYER 



JOS 



GET ACTION ID 
FROM ACTION LIBRARY 



710 

/ 



* CALL ACTION BY ID (ACTION ID) 



712 

y 



ACTION ! 



.-J 



Figure 7 



P"T/CN 0 10 8 7 



J 



8/9 




Figure 8 



P^T/CN 03/ 0 10 8 




9/9 



300 



MAKER 



900 



~1 



DEFINE THE PAGE 
GLOBAL VARIABLE 
AND FUNCTIONS 



SC4 



DEFINE THE SCRIPT 
FUNCTION FOR AN 
OBJECT 



JL 



SOS 



PACKAGE THE LINKAGE 
INFORMATION AND SCRIPT 
MODULE AND STORE 
TO THE ACTION LIBRARY 



902 

DIM PLAYFLAG 

sub playvideo () 
player.playvideo 0 

END SUB 
SUB Pi-AYAUDl 0 
PLAYER,PL.AYAUDl () 
END SUB ^ 

906 



\ /sub OBJECTNAME (INT X. IN I Y) 
\_J IF PLAYFLAG < 0 THEN 



PLAY^/]DEO () 

ELSE 
PLAYAUDIO 0 
END SUD 



51S 



OBJ 1 -> SCRIPT 1 
OBJ 2 SCRIPT 2 



y 302 

910 1 
/ 



LOAD THE SCRIPT 
MODULE FOR THE 
CURRENT PAGE FROM 
THE ACTION LIBRARY 



T 



912 



ADD THE SCRIPT MODULE 
TO THE SCRIPT ENGINE 



CALL THE SCRIPT 
FUNCTION 



PLAYER 



^18 



ACTION ! 




Figure 9 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 

BEST AVAILABLE IMAGES 

Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SmES 
JSTfADED TEXT OR DRAWING 
P'^LURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GR^ SCALE DOCUMENTS 

□mLINES or MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



